Summary and Setup

General Information

This lesson is an introduction to using high-performance computing systems effectively. We obviously can’t cover every case or give an exhaustive course on parallel programming in just one lesson of teaching time. Instead, this lesson is intended to give students a good introduction and overview of the tools available and how to use them effectively.

By the end of this lesson, students will know how to:

Checklist
  • Identify problems an HPC system can help solve
  • Use the UNIX shell (also known as terminal or command line) to operate a computer, connect to an HPC system, and write simple shell scripts.
  • Submit and manage jobs on an HPC system using a scheduler, transfer files, and use software through environment modules.

Schedule

Time Item
09:00 – 10:00 Connecting to the remote HPC system
10:00 – 10:20 Why use an HPC System?
10:20 – 11:00 Working on a remote HPC system
11:00 – 11:30 Tea/coffee break
11:30 – 13:00 Working with the scheduler
13:00 – 14:00 Lunch
14:00 – 14:45 Accessing software via Modules
14:45 – 15:30 Transferring files with remote computers
15:30 – 16:00 Tea/coffee break
16:00 – 16:40 Using resources effectively
16:40 – 17:00 Using shared resources responsibly
17:05 – 18:00 Career development discussion for female participants
18:00 – 19:00 Free time
19:00 – Event dinner (The Nag’s Head)
Prerequisite

To begin this lesson you must have completed the following:

  • Participants must bring a laptop with a Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) that they can download and run a terminal application on. Alternatively, they should have a few specific software packages installed (listed in the Setup section below).
  • Participants must agree to abide by the ARCHER2 Code of Conduct
  • Attendees should be familair with using the bash shell. This is covered in the Introduction to the Shell HPC course
  • Complete the software/system setup including how to connection to ARCHER2 via SSH.

Quick-Start: Setup

There are several pieces of software you will wish to install before the workshop. Though help will be provided at the workshop, we recommend taking a look at the below information to help you arrive prepared!

Prerequisite

Bash and SSH

This lesson requires a terminal application (bash, zsh, or others) with the ability to securely connect to a remote machine (ssh).

Quick-Start: Shell & SSH Set-up

The shell is a program that enables us to send commands to the computer and receive output. It is also referred to as the terminal or command line.

Alongside this, all attendees should have an SSH client installed. SSH is a tool that allows us to connect to and use a remote computer as our own.

Please follow the directions below to install a Shell and SSH client for your system if you do not already have one.

Discussion

Details

Some computers include a default Unix Shell program. The steps below describe some methods for identifying and opening a Unix Shell program if you already have one installed. There are also options for identifying and downloading a Unix Shell program, a Linux/UNIX emulator, or a program to access a Unix Shell on a server.

Computers with Windows operating systems do not automatically have a Unix Shell program installed. In this lesson, we encourage you to use an emulator included in MobaXterm for Windows, which gives you access to both Bash shell commands and SSH.

Once installed, you can open a terminal by running the program MobaXterm from the Windows start menu.

Reference

Discussion

Alternatives to MobaXterm and Git for Windows

Other solutions are available for running Bash commands on Windows. There is now a Bash shell command-line tool available for Windows 10. Additionally, you can run Bash commands on a remote computer or server that already has a Unix Shell, from your Windows machine. This can usually be done through a Secure Shell (SSH) client. One such client available for free for Windows computers is PuTTY. See the reference below for information on installing and using PuTTY, using the Windows 10 command-line tool, or installing and using a Unix/Linux emulator.

For advanced users, you might choose one of the following alternatives:

Warning:

Commands in the Windows Subsystem for Linux (WSL), Powershell, or Cygwin may differ slightly from those shown in the lesson or presented in the workshop. Please ask if you encounter such a mismatch — you’re probably not alone.

On macOS, the default Unix Shell is accessible by running the Terminal program from the /Application/Utilities folder in Finder.

To open Terminal, try one or both of the following:

  • In Finder, select the Go menu, then select Utilities. Locate Terminal in the Utilities folder and open it.
  • Use the Mac ‘Spotlight’ computer search function. Search for: Terminal and press Return.

macOS comes with SSH pre-installed, so you should not need to install anything. You can type ssh --help to test if it is available.

Reference

How to Use Terminal on a Mac

On most versions of Linux, the default Unix Shell is accessible by running the (Gnome) Terminal or (KDE) Konsole or xterm, which can be found via the applications menu or the search bar.

Linux comes with SSH pre-installed, so you should not need to install anything. You can type ssh --help to test if it is available.

If none of the options above address your circumstances, try an online search for: Unix shell [your operating system].

Quick-Start: Connect to ARCHER2

For this lesson, we will be using ARCHER2, the latest iteration of the UK National Supercomputing Service. ARCHER2 will be available to you for the duration of the course and for a few days afterwards, to allow you to complete the practical exercises and put some of what you have learned into practice.

Please register for an ARCHER2 account.

Callout

Note: If you already have an ARCHER2 account you can use that for this course. Please see see the notes at the end of this page rather than requesting a new account. Of course, if you prefer to sign up for another ARCHER2 account specifically for this course, you are welcome to do this instead!

Sign up for a SAFE account

To sign up, you must first register for an account on SAFE (our service administration web application):

If you are already registered on the SAFE you do not need to re-register. Please proceed to the next step.

  1. Go to the SAFE New User Signup Form
  2. Fill in your personal details. You can come back later and change them if you wish. Note: you should register using your institutional or company email address - email domains such as gmail.com, outlook.com, etc. are not allowed to be used for access to ARCHER2
  3. Click “Submit”
  4. You are now registered. A single use login link will be emailed to the email address you provided. You can use this link to login and set your password.

Sign up for an account on ARCHER2 through SAFE

  1. Login to SAFE
  2. Go to the Menu “Login accounts” and select “Request login account”
  3. Choose the ta215 project and click “Next”
  4. On the next page, the ARCHER2 system should be selected. Click “Next”

Now you have to wait for the course organiser to accept your request to register. When this has happened, your account will be created on ARCHER2. Once this has been done, you should be sent an email. If you have not received an email but believe that your account should have been activated, check your account status in SAFE which will also show when the account has been activated. You can then pick up your one shot initial password for ARCHER2 from your SAFE account.

Generate an SSH key pair and upload it to SAFE

In addition to your password, you will need an SSH key pair to access ARCHER2. There is useful guidance on how to generate SSH key pairs in the ARCHER2 documentation, but we will also cover this in the first lesson.

Once you have generated your key pair, you need to add the public part to your ARCHER2 account in SAFE:

  1. Login to SAFE
  2. Go to the Menu “Login accounts” and select the ARCHER2 account you want to add the SSH key to
  3. On the subsequent Login account details page click the “Add Credential” button
  4. Select “SSH public key” as the Credential Type and click “Next”
  5. Either copy and paste the public part of your SSH key into the “SSH Public key” box or use the button to select the public key file on your computer.
  6. Click “Add” to associate the public SSH key part with your account

The public SSH key part will now be added to your login account on the ARCHER2 system.

Configure TOTP passwords

Archer2 now uses one time passwords (TOTP) for multi-factor authentication (MFA). One time passwords are a common security measure used by banking, cloud services and apps that create a changing time limited code to verify your identity beyond a password and username.

To setup your MFA TOTP you will need an authenticator application on your phone or laptop. Follow the steps at the SAFE documentation, ensuring you create the code for your ta215 project account.

You will only be prompted at login for your TOTP code once a day.

Log into ARCHER2

You should now be able to log into ARCHER2 by following the login instructions in the ARCHER2 documentation.

Callout

Using an existing ARCHER2 account

If you wish to use an existing ARCHER2 account for the course, that is perfectly fine. The only differences from the specific course account are that you will not have access to the course account code. Instead, you can use your existing account code (the cost of the jobs is negligible as they are very short and small).